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Mathematics education researchers conducting instruction experiments 
using a design research methodology are challenged with the analysis of 
often complex and large amounts of qualitative data. In this paper, we 
present two case studies that show how multimedia analysis software can 
greatly support video data analysis and theory development in design 
research. The software can (a) act as a type of mould for organising large 
amounts of data; (b) contribute to improving the trackability and reliability 
of the research; and (c) support theory generation and validation. We 
propose an integrated model that elucidates the complex process of data 
analysis by showing how each of the components that are involved in the 
data analysis procedures feeds into the emerging local instruction theory. 
The model combines the intricate cycles of coding and analysing raw video 
data with the cumulative cyclic process that characterises design research in 
mathematics education. Our experiences with this model may support 
other mathematics education researchers in the development of thorough 
and empirically supported local instruction theories from complex 
qualitative analyses. 


Design research in mathematics education has both a theoretical and an 
applied purpose: theoretical in terms of coming to an understanding of 
mathematical thinking, teaching and learning, and applied in terms of using 
this understanding to improve mathematics instruction (Schoenfeld, 2000). 
As a result, researchers do not settle for answers that merely illustrate what 
works best. Instead, they use methods that both aim to understand why as 
well as how mathematics instruction works in practice (cf. Freudenthal, 1983; 
Gravemeijer, Bowers, & Stephan, 2003). 

What characterises the process of data analysis in design research is the 
interaction between data analysis and theory development. At a practical 
level, this interaction is illustrated in Jacobs, Kawanaka, and Stigler's (1999) 
cyclical process of analyzing video data. Oriented more towards theory 
development, Gravemeijer (2004) describes design research in terms of a 
cumulative cyclic process. In this paper, we propose to embed Jacobs et al.'s 
(1999) model into Gravemeijer's (2004) model for a more encompassing 
representation of the intricate interactive processes that are essential for 
theory development in design research. The significance of this integrated 
model is that it encompasses the steps that are necessary for keeping track of 
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the process of analysing raw qualitative data for theory development in 
design research. 

The challenge of qualitative mathematics education research is that the 
investigations often entail a large amount of data that derive from complex 
educational settings (cf. Gravemeijer & Cobb, 2006). Multimedia data 
analysis software is therefore a welcome tool for analysing raw qualitative 
data (Powell, Francisco, & Maher, 2003; Stigler, Gallimore, & Hiebert, 2000). 
This allows for the interactive and automatic coding of rich text, image, 
audio, and video materials. After importing raw data in the form of, for 
example, a video, screenshots, or scans of written work into the program, the 
researcher can organise the data by segmenting the data into "quotations" 
(i.e. video clips or "meaningful chunks"; Stigler et al., 2000), adding 
comments to the quotations, creating codes to label the quotations, and 
linking the appropriate codes to specific quotations. 

In this paper, we argue that the use of multimedia data analysis 
software can help to conduct a cyclical analytical process in design research 
as implied by the integrated model that we propose. This is valuable for 
warranting the validity and reliability of conclusions that are drawn from 
raw qualitative video data. We discuss how multimedia data analysis 
software can be used to cultivate the interaction that is inherent to observing 
and analysing qualitative design research data. Furthermore, we present 
several benefits of using multimedia data analysis software for organising 
the complex raw qualitative data and for supporting the quality of the 
research. 

To illustrate the benefits of using multimedia data analysis software, we 
present the MENS research project (Van Nes & De Lange, 2007; Van Nes & 
Doorman, 2009) and the ToolUse research project (Drijvers, Doorman, Boon, 
Van Gisbergen, & Gravemeijer, 2007). These are two examples that bring the 
integrated model into practice. We describe how the multimedia data 
analysis software ATLAS. ti 1 supported the organisation of and connection 
between different sources of qualitative data, how it encouraged the sharing 
and discussing of the research with others, and how this contributed to the 
reliability and trackability of the research. The interaction between data 
analysis and the evaluation and subsequent adjustments of conjectures is 
what stimulated theory development in both projects. This contributed to 
bridging practice with theory for drawing valid and reliable conclusions. 


Designed and produced by ATLAS.ti GmbH in Berlin 
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Brief Introduction to Design Research in Mathematics 

Education 

Design research in mathematics education is oriented towards 
generating empirically based theories. It was first described by Glaser and 
Strauss (1967) and further developed into a methodology that is 
characterised by the cumulative, cyclical process of theory development 
(e.g., Freudenthal, 1973; Gravemeijer, 1994; Cobb, diSessa, Lehrer, & 
Schauble, 2003). This cyclical process involves an iterative procedure of 
theory-driven adjustments to the intervention and amendments to the 
hypotheses, which, together, are intended to lead to an improved and 
empirically supported theory. 

Realistic Mathematics Education (RME) inspires the development of 
mathematics education by offering a pedagogical and didactical perspective 
on mathematical teaching and learning. Mathematics in RME is seen as a 
"human activity" that is driven by the act of mathematising (Freudenthal, 
1973, 1991). Mathematising means more than gaining mathematical 
knowledge and becoming adept at performing mathematical operations. 
Rather, it involves understanding underlying mathematical abilities such as 
ordering, classifying, generalising, and formalising (Treffers, 1975). 
Freudenthal (1973, 1991) posits the importance of guided reinvention for 
stimulating mathematisation. This principle underlies the construction of 
knowledge as if the student "reinvented" it. It differs from the discovery 
perspective (which states that symbolic systems exist and that children must 
try to understand them with help from adults), because in the (re) invention 
perspective, the children really (re)invent the mathematics by being placed 
in situations that require mathematising (Gravemeijer, 1998). Flence, the 
designer's role in this "active learning process" is to encourage the children's 
spontaneous strategies without imposing the mathematical knowledge on 
them. It is the task of the designer to offer children tools with which to build 
upon earlier knowledge for constructing and internalising new insights. 
Together, this is implied by the term "guided". 

In practice, guided reinvention means that the teacher can anticipate the 
types of mathematical thinking that children may develop at particular 
moments, and provide the concepts, models, and symbols when the children 
may find them necessary. At a classroom level, the designer must help the 
children converge their mathematical ideas with each other and with the 
teacher so that a shared mathematical standard can be achieved. To achieve 
this, (re)invention includes the introduction of concepts, models, and 
symbols that are directly related to what is necessary for the children's 
learning and understanding at that particular moment, and ideally builds 
upon the children's present knowledge. 
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Freudenthal (1991) used the term "instruction experiment" to refer to a 
research design that includes an instructional sequence that is meant to 
broaden the children's insight into a particular mathematical construct, 
while it, at the same time, provides the researchers with a greater 
understanding of the children's learning processes. The aim of developing 
such an instruction experiment, then, is not necessarily to conclude that the 
instructional sequences teaches the children about a particular mathematical 
construct, but more to come to an analysis about how this instructional 
sequence may or may not have stimulated the children's thinking 
(Gravemeijer et al., 2003). 

Theory development in design research concerns the generation of local 
instruction theories (Gravemeijer, 1994). This theory is a learning trajectory 
based on mathematical, psychological, and didactical insights about how 
researchers expect the children to progress towards an aspired level of 
reasoning. In practice, it encompasses an instructional sequence, a 
description of the coinciding learning processes, a description of the 
classroom culture, and an analysis of the proactive role of the teacher. The 
progression should take into account both the cognitive development of the 
individual students, as well as the social context (i.e., the people involved, 
classroom culture, and type of instruction) in which the instruction 
experiment is to take place (Cobb & Yackel, 1996). This implies that the 
outcomes of the research should not be generalised beyond this particular 
setting. Instead, the local instruction theory is intended to inspire researchers 
to implement the instruction experiment in other settings as a way to 
evaluate the local instruction theory and contribute to the development of a 
more encompassing theory. 

The cyclical process that characterises design research is illustrated in 
Figure 1. To come to a local instruction theory, researchers define a 
hypothetical learning trajectory that shapes the design of each instructional 
activity in the instructional sequence (Simon, 1995). Researchers then 
implement an instructional sequence in an "instruction experiment," 
perform retrospective analyses on the transcripts from these lessons, adjust 
the hypothetical learning trajectory accordingly during a "thought 
experiment," and improve the instructional sequence to cohere with the 
amended hypotheses and conjectures about successful patterns and the 
ways to support them. Finally, the procedure is repeated by entering a new 
cycle of implementing the new instructional sequence and learning from the 
class-experiences during a thought experiment. 
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Figure 1. The cyclical procedure involved 
in design research (Gravemeijer, 2004). 


The type of data that researchers collect in an instruction experiment 
should help them gain insight into behavioral patterns that may confirm or 
refute their hypotheses and contribute to answering the research questions 
(Gravemeijer & Cobb, 2006; Powell et al., 2003). However, the analysis of 
such qualitative data is usually more comprehensive than the statistical 
manipulation of quantitative data. The often large amounts of qualitative 
data have to be reduced and transcribed to become manageable before any 
analysis can take place. Strauss and Corbin (1998, p. 13) underlined the need 
to organise the raw data in qualitative data analysis, since the process of 
organising itself contributes to: (a) building rather than testing theory, (b) 
providing researchers with analytic tools for handling masses of raw data, 
(c) helping analysts to consider alternative meanings of phenomena, (d) 
being systematic and creative simultaneously, and (e) identifying, 
developing, and relating the concepts that are the building blocks of theory. 

Given the small-scale and often exploratory type of studies in qualitative 
design research, it is also challenging to guarantee external validity of 
emerging (theoretical) conjectures (Jacobs et al., 1999). This is because 
qualitative analysis "draws on both critical and creative thinking" and relies 
on "the interplay between researchers and data" (Strauss & Corbin, 1998, p. 
13). In theory, qualitative researchers may approach external validity by 
enlarging their pool of data, but in practice, the time-consuming data 
collection and analyses often place limits on the size to which the study can 
grow. Consequently, design researchers work towards more reliable results 
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through, for example, triangulation, combining data sources and 
establishing the trackability of their conjectures (Gravemeijer & Cobb, 2006; 
Smaling, 1987). This requires an as thorough and exhaustive as possible 
documentation of the observations, considerations, decisions, and changes 
that the researchers make to the instructional sequence, to their conjectures, 
and, ultimately, to the local instruction theory, as they progress along the 
cycles of design research. In the section below, we suggest how such 
analyses may be enhanced through integrating Jacob et al.'s (1999) cyclical 
analytical model with Gravemeijer's (2004) cumulative cyclic process of 
design research. 


Merging Two Models 

Jacobs et al.'s (1999) cyclical analytical process (see Figure 2) integrates 
qualitative and quantitative approaches for the analysis of video data. This 
cyclical analytical process summarises how the researchers initially "watch 
and discuss" the videos and subsequently "generate hypotheses" about 
what they are discovering. Their goal is to ultimately "transform the video 
images into objective and verifiable information" (Jacobs et al., p. 718). Such 
video data allow for a unique iterative process of data analysis that 
otherwise depends on a complicated comparison of data that has to be 
collected linearly (Jacobs et al.). 


Watch and discuss 



Linlc to video 
Analyze, interpret 



Generate hypothesis 
Develop code 


- Apply code 



Figure 2. Model for the cyclical analytical process of analysing video data 

(Jacobs et al., 1999) 


While Jacobs et al.'s (1999) model for the cyclical analytical process of 
analysing video data points to an iterative process of data analysis. 
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Gravemeijer's (2004) model of design research analogously describes 
cumulative iterative cycles that illustrate the interaction between theory and 
practice. Both models focus on interaction as a way to propel the research 
towards a local instruction theory. Thus, we propose to embed the specific 
process of analysing video data as described by Jacobs et al.'s model (Figure 
2) into the cyclical process of interpreting instruction experiments in light of 
the overall research objectives as described by Gravemeijer's cycles of design 
research (2004, Figure 1). The integrated model is depicted in Figure 3. 



Instructionexperiment Instructionexperiment 


Instruction experiment 


Figure 3. The proposed integration of Jacobs, Kawanaka, and Stigler's (1999) 
model for the cyclical analytical process of analysing video data into 
Gravemeijer's (2004) model of the cumulative 
cyclic process of design research. 

The integrated model highlights the process of data analysis that occurs 
immediately after performing one instruction experiment and just before the 
start of the next instruction experiment in design research. In this phase, the 
data that is collected in the first instruction experiment is analysed and the 
patterns in the data are used to reassess the researcher's hypotheses, the 
design of the instruction experiment, and, ultimately, the emerging local 
instruction theory (Gravemeijer, 2004). This is how the model adheres to 
Gravemeijer's cycles of design research. 

To describe the process of data analysis that occurs inbetween two 
instruction experiments at a more practical level, we turn to Jacobs et al.'s 
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model (1999). The analysis of video data involves iterative cycles of 
organising, examining, and coding the data. This cyclical process contributes 
to the development of the overarching theory of the research by welcoming 
quantitative findings as a supplement for interpreting the results. This leads 
to new research questions and it helps to refine codes and locate footage that 
can support the researchers' conjectures (Jacobs et al.). In Gravemeijer's 
(2004) terms, such an overarching theory denotes the emerging local 
instruction theory in mathematics education research. 

What links the two models in our proposed model together, then, is that 
the components that make up the data analysis procedure (organising, 
examining, and coding) are not only reflexively related, but can also feed 
into the emerging local instruction theory at any of the steps in the analysis. 
This interaction is what characterises the process of theory development in 
design research in mathematics education (Gravemeijer & Cobb, 2006). As 
such, the integrated model shows how Jacobs et al.'s (1999) analytical 
process for "watching, coding and analyzing the data, with the goal of 
transforming the video images into objective and verifiable information" (p. 
718) can be applied to design research to enhance the value of conclusions 
that are drawn from it. 

Analysing Raw Qualitative Data 

The integration of the cyclical process of analysing video data as 
described by Jacobs et al. (1999) with the design research cycles as described 
by Gravemeijer (2004) provides more insight into the use of multimedia 
software analysis programs for elucidating the intricate and interactive 
process that is involved in generating questions and theories in analysing 
raw qualitative data from specifically design research in mathematics 
education. Powell et al. (2003) extensively reviewed literature on the role of 
videotape data in mathematics education research. This has led them to 
distinguish a sequence of seven interacting nonlinear phases in an analytical 
model for studying the development of learners' mathematical ideas. These 
are viewing the video data attentively, describing the data, identifying 
critical events, transcribing, coding, constructing a storyline, and composing 
the narrative (p. 413). Powell et al.'s analytical model for studying 
mathematical learning using video data recognises that research on the 
development of mathematical thinking "is not a linear, unidirectional 
process" (p. 432). Hence, the phases that they distinguish must be 
interpreted as a "cyclic and recursive process that requires multiple visits 
with data as well as collaborative exchanges with researchers with different 
analytical and theoretical foci" (p. 432). In this way, they suggest that the 
model may be "a step forward in developing an analytical approach using 
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video data for observational investigations into the development of learners' 
mathematical ideas and forms of reasoning" (p. 432). 

In contrast to how Jacobs et al. (1999) highlight the search for 
explanations and generating hypotheses, Powell et al. (2003) put a relatively 
greater accent on the search for patterns in the data. In support of Powell et 
al.'s model, we note that multimedia data analysis software is an important 
tool in design research for bridging observations from educational practice 
to theory. This advantage can be described in terms of several benefits. First, 
multimedia data analysis can relieve researchers of the cognitive strain that 
may be associated with trying to keep track of all the thoughts and 
observations that could be shaping the conjectures and contributing to the 
theory. ATLAS, ti can act as a mould for configuring and organising the often 
large amounts of raw qualitative data into quotations that mark critical 
events (Stigler et al., 2000). Once the first set of quotations is defined, the 
researchers can add comments to the quotations and create storylines with 
analytical notes. The researchers can tailor and subsequently amend the 
length of the quotations according to what seems to fit best for answering 
the original and emerging research questions. Subsequently, the researchers 
can refine the units of analysis and define codes to describe the quotations in 
any order and with any frequency that may be necessary for identifying the 
importance of the episode for the developing theory. The codes can further 
be combined into "families" as a way to organise their hierarchical structure. 

After organising the raw video data into quotations with corresponding 
codes, the researcher can play a specific clip from the resulting list of 
quotations that are part of one particular video, look up the (network of) 
codes that are associated with the clip, and create outputs that summarise 
the details of the clip, the type and number of codes that are linked to the 
clip, or, more generally, the frequency with which particular codes are 
linked to the entire video. This video data may be supplemented with other 
data sources, such as scans from written student materials, which can be 
imported and coded in the program. Further, the researcher can define 
queries to search for specific patterns and frequencies in how the quotations 
are coded. These features of the program allow for the researcher to spiral 
into particular episodes as a means to generate theory in an organised way. 
The software helps to make the raw data more accessible for analysis 
because it supports the organisation of the raw data and correlated 
reflections and ideas of the researcher. 

A second benefit of using a multimedia data analysis tool for video data 
analysis is that it can supplement qualitative analyses with quantitative data. The 
frequency counts for coding the quotations can become the subject of 
quantitative analyses (Jacobs et al., 1999). Although much of the data may be 
gathered according to qualitative research traditions, the frequency counts in 
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the tool give it the potential to bridge with larger-scaled statistical analyses. 
Assuming that the data adheres to the strict conditions that are required for 
statistical procedures, the researcher can export the counts to other statistical 
software, run statistical analyses, and visually represent the outcomes. The 
outcomes of such quantitative analyses can then be interpreted and linked 
back to the videos to reintegrate the quantitative findings with the 
qualitative interpretations. 

Next to organising the raw data and making it accessible to both 
qualitative and quantitative analyses, a third benefit of using multimedia 
data analysis software is that it can help to improve the trackability of 
inferences that are drawn in the data analyses. The quotations, for instance, 
are convenient to refer to when discussing conjectures or conclusions with 
others (Stigler et al., 2000). They make it easier to transcribe and present 
particular instances of one video or of a combination of data sources to 
illustrate one specific inference or to find patterns in the data. The researcher 
can also make reference to specific quotations and codes while 
communicating the analyses in a presentation or a written report. This 
makes it accessible and attractive for others to trace the researchers' 
argumentation. This makes it convenient to share the raw data with 
colleagues and to, for example, determine interrater reliabilities by 
comparing how each researcher segmented the videos and defined and 
assigned the codes to the quotations. 

Taking these three benefits of multimedia data analysis software 
outlined above together, it becomes clear that working with such a tool 
contributes to theory generation and validation for theory development in 
design research. This relates to an essential interaction between the 
qualitative data and the analyses that shape conjectures about data. In what 
follows, we describe two research projects that put the integrated model into 
practice and that illustrate the role of multimedia data analysis software 
ATLAS. ti in theory development of design research. 

The Role of ATLAS.ti in Two Research Projects 

In this paper, we make use of two examples of design research to 
illustrate the role of multimedia data analysis software in developing a local 
instruction theory. Both examples involve an instruction experiment that is 
designed to inspire, test, and retest hypothetical learning trajectories about 
how the children are expected to acquire a particular mathematical construct 
as they work through the instructional sequence. The hypothetical learning 
trajectories that coincide with this instructional sequence ultimately 
contribute to the development of a local instruction theory. Hence, the 
aspired outcome of the instruction experiment in each project is a local 
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instruction theory that contributes to insight into the learning processes of 
children in a particular setting and for a particular mathematical construct. 

The MENS Project 

The first example is the Mathematics Education and Neurosciences 
(MENS) project. This is an interdisciplinary investigation into the 
development of spatial thinking and number sense of young children (Van 
Nes & De Lange, 2007; Van Nes & Doorman, 2006, 2009). We turn in this 
paper to the mathematics education component of the project. The 
researchers are building upon studies by Battista and Clements (1996) and 
Battista, Clements, Arnoff, Battista, and Van Auken Borrow (1998) as well as 
by Mulligan, Mitchelmore, and Prescott (2005), and Mulligan, Mitchelmore, 
Martson, Highfield, and Kemp (2008), which have highlighted the 
importance of spatial structuring for the development of early mathematical 
abilities. Examples of spatial structures that are considered relevant to the 
development of kindergartners' mathematical abilities are finger patterns 
and domino-dot configurations. It is essential for children to gain insight 
into spatial structures so that they can learn to abbreviate basic numerical 
procedures and to develop more sophisticated numerical strategies 
(Anghileri, 1989; Van Eerde, 1996). Hence, the main conjecture is that 
children's early insight into and use of spatial structures can stimulate the 
development of their emerging numerical abilities. This conjecture is 
supported by studies showing how children with numerical difficulties 
often do not develop counting strategies beyond unitary counting 
procedures (Mulligan et al., 2005; Pitta-Pantazi, Gray, & Christou, 2004). 

To gain insight into how kindergartners' spatial structuring ability may 
be related to their emerging number sense, the researchers designed an 
instructional sequence to better understand how kindergartners can be 
encouraged to abbreviate numerical procedures by recognising and making 
use of spatial structures. This sequence consisted of five lessons that were 
implemented in two subsequent rounds in the instruction experiments after 
the first and before the second interview. The aim of this instruction 
experiment was to (a) better understand how children learn to make use of 
spatial structures to abbreviate mathematical procedures, and (b) highlight 
ways to amend the instructional sequence so that the activities in the lessons 
interweave with children's level of mathematical understanding and 
promote spatial structuring to support kindergartners' number sense. After 
performing the first round, the researchers analysed the instructional setting, 
adjusted the instructional sequence accordingly, and tried out the revised 
instructional sequence in the second round. 

The children were video-taped during the interviews and the classroom 
instruction activities. This resulted in a total of almost 50 hours of video 
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recordings. Hence, the first challenge for the researchers was to organise this 
raw data into a format that would help make sense of how the children were 
solving the problems; how the children were developing in their 
understanding; how the researcher, the teachers, and the instructional 
activities had played a role in this development; and how proactive 
individual and classroom instruction could ultimately support the children's 
learning. We will discuss how the multimedia data analysis program 
ATLAS. ti provides a format that proved to be essential for organising and 
making sense of all the data. 

The TooIUse Project 

The second example that we discuss in this paper is the TooIUse Project. 
The TooIUse project concerns the use of computer tools for grade 8 (13-14 
years old) students' acquisition of the mathematical concept of function, in 
an effort to identify the relationship between the use of technological tools 
and the learning of mathematics (Drijvers et al., 2007). 

As in the MENS project, the research design of the TooIUse Project 
involves a cyclic process of instructional design, instruction experiments, 
and data analysis (Gravemeijer, 2004). Teaching materials and a computer 
tool were developed in the design phase of the project. This computer tool, 
"Algebra Arrows," is meant to support the construction of input-output 
chains of operations. The applet is embedded in an electronic learning 
environment and the researchers, as well as the teacher, can monitor and 
compare the students' responses with class results. The activity of 
investigating this applet in pairs was expected to foster a learning process of 
"trial-and-improve" and "negotiation-of-meaning." During these 
investigations, students could find patterns and develop strategies while 
dealing with dynamic input-output dependencies. As such, the computer 
tool was expected to provide students with a means to structure inquiry 
about these input-output dependencies through constructing and modifying 
arrow chains and through generating and comparing argumentations. The 
aim of the project is to understand how computer tools can be integrated in 
an instructional sequence on the function concept so that their use can foster 
learning. 

The instruction experiments were conducted with three grade 6 classes 
at three different schools. The children were video-taped during each 
teaching session and group work (17 hours), and screen videos of three pairs 
of students working with the computer were collected (6 hours). Students' 
final answers to the activities were saved on a central server (41 pairs of 
students). In addition, the researchers collected students' written work and 
the results of a written assessment at the end of the instruction experiment. 
The data was qualitatively analysed and coded using ATLAS. ti. Ultimately, 
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the outcomes of this research may contribute to improving the instructional 
sequence and to preparing a second instruction experiment for finding 
empirical support for the conjectured process of teaching and learning of the 
function concept using computer tools. 

Together, the analysis procedures in the MENS project and in the 
ToolUse project illustrate the cyclic process of data analysis and theory 
development that is inherent to design research. In what follows, we discuss 
how the analysis procedures in both projects benefited from the use of 
multimedia data analysis software. 

Multimedia Data Analysis and Theory Development 
in the MENS and ToolUse Projects 

The MENS project and the ToolUse project serve as illustrations for the 
benefits of using multimedia data analysis software that we described 
above: organising the raw qualitative data, contributing to the reliability of 
the research, and making the data more accessible to others. This has 
supported both projects' theory development and it embodies the integrated 
model that was proposed in the section regarding the analysis of the 
qualitative data. Despite this convenient use of ATLAS. ti as a tool to 
organise the raw data, it took the researchers in both projects a considerable 
amount of time that should not be ignored. We will reflect on this 
complication in the conclusion of this paper. 

Organising Data 

For the analyses of the instructional sequence in the MENS project, the 
researchers created quotations of any events and occurrences that would 
give them indications of (a) how the children were learning from the 
instructional setting, or, otherwise, why no learning occurred, (b) how the 
teacher was guiding the lesson and stimulating the children's learning, (c) 
what revisions would be necessary to improve the instructional setting, and 
(d) any instances that could contradict the hypothesis, to control for any self- 
fulfilling prophecies. As the researchers spent more time watching and 
discussing the quotations, the quotations became increasingly more refined 
to highlight one particular aspect of an instructional activity or an 
interesting remark or expression. 

In the process of creating the quotations of the instruction experiment, it 
initially seemed most useful to code the quotations with the names of the 
people who were involved. In this way, each of the quotations could be 
identified according to who was involved in that particular event. As the 
researchers were discussing the quotations, however, they realised that, 
although this was a useful way of organising the complex raw data, it 
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hampered the generation and validation of theory. This is because 
separating the output into each individual gave the false impression that 
insight into the children's learning processes required only data from 
separate individuals. One important component of a local instruction theory, 
however, is a description of the overarching instructional setting, which 
includes not only the individual children, but also the teacher, the 
instructional activity and materials, and the interaction between the children 
(Cobb & Yackel, 1996). 

The researchers' second approach to organising the raw data took more 
of the social setting of the instruction experiment into account. The codes 
were still pre-categorised into "child," "teacher," and "researcher," but 
within these categories, the coding occurred more inductively; as the 
researchers were observing the quotations, they analysed the situations 
against the background of the research questions, and defined codes that 
could encompass the notable events that they were flagging. An example of 
such a code is "teacher: makes external link," indicating that the teacher 
made reference to, for instance, a spatial structure that the class had 
discussed in a previous activity. Another example is "child: overt moment of 
insight," which refers to an "aha-moment" when a child explicitly 
communicates a new understanding. 

The inductive and reflexive process of watching the videos and defining 
relevant codes illustrates the interaction that is key to theory development in 
design research (Gravemeijer et al., 2003). This interaction is also central to 
Powell et al.'s (2003) analytical model for studying mathematical learning, as 
well as Jacobs et al.'s (1999) cyclical analytical model for data analysis. As 
the researchers in the MENS project were analysing each quotation, defining 
relevant codes, applying the codes to the quotations, and sometimes 
redefining the codes, they developed a list that was as exhaustive as possible 
to encompass the behavioural patterns that the researchers conjectured to be 
essential for attaining the research objectives. 

Yet, this thorough organisation of the raw data into quotations with 
codes still marks only the beginning of the analytical process. During the 
coding phase, the researchers were also discussing the relative contribution 
of the codes for answering the research questions. From these discussions it 
became apparent that some codes deserved more attention than other codes. 
For example, the code for "child: uses spatial structuring" is more relevant 
to the research than the code for "child: is distracted." At the same time, the 
discussions about the relative contributions of the codes led the researchers 
to begin to use memos as a way of keeping track of their thoughts and 
reasoning. An example of a memo is "a missed chance," where the 
researchers noted that the teacher seemed to have missed the opportunity to 
stimulate the child in a particular quotation. 
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As more and more memos were added to the list, there also emerged 
relative differences in the types of memos that were defined. Not only could 
the researchers relate and physically connect more memos to individual 
quotations, they could also connect more memos to other memos. Groups of 
memos inspired the researchers to think about memos at a higher, more 
theoretical level. A diagram of this network helped to define and focus on 
the physical and theoretical connections between clips and conjectures that 
emerged during the process of organising and making sense of the data (see 
Figure 4). The network not only physically pictured the codes that were 
linked to a quotation (i.e., nodes with connecting lines), but also sometimes 
made the nature of the link apparent. A code such as "C: Focusgroup," 
concerning a child ("C") from the focus group, was connected to a code such 
as "C: Grade 2" by the summarising link "is part of." 



C: Gives a verbal explanation 

{l-0}~ 

The child responds verbally 


► 


R: Encouragement {1-0} I 


> 


$ R: Asks a continuing question 

{1-0} 


* 


U R: Seeks attention {1-0} I 


U C Focusgroup {1-1}~ 

The child is part of the focus group 


1 * \ 

is part of 

/' \ , 

/|$C:Grade2{l-l}~ 

' The child who is responding is in Grade 2 




' a V* i i i * k y 



[2:83] Activiteit 2_De Truqesdoos.wm.. 


Developing the use of structures 

Variety of levels in children's spatial 
structuring ability indicate a general 
developmental progression. Development 
appears in the instruction activities, 
determining the child's approach and 
success at attempting the activities. 

* 

£3 

Teacher elucidates the structure 

Example: pulling the blocks apart, 
nonverbal explanations, comparing types 
of structures 


Jenna gives a basic explanation 

It is 7 by how you count it'. Shows 
Jenna's level of working with quantities. 
' * She keeps counting one by one . No 
indication of spatial structuring, more of 
unitary counting. 


Figure 4. An example of a network of nodes picturing the codes, theory-type, memo-type, and commentary-type memos 

that are attached to quotation 2:83. 
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To denote the differences between types of memos, the researchers 
redefined the memos into a hierarchical structure of commentaries, memos, 
and theories. Most of the memos became commentaries, which were defined 
as incidental observations that may be interesting for theory development, 
but that were linked to only one particular observation (i.e. one quotation). 
A commentary in Figure 4 is "Jenna gives a basic explanation." This refers to 
a particular child who, on one occasion (i.e., this quotation), seemed more 
focused on unitary counting rather than on structuring. 

One type of memo above commentaries is memos in the broader sense 
of the term. These memos came to denote episodes that can directly be 
related to children's spatial structuring and the role of the instructional 
sequence, which may provide essential clues for theory development. An 
example of a memo is "Teacher elucidates the structure," alluding to the role 
of the teacher in stimulating children's insight into spatial structures. Higher 
up in the hierarchy are theories, which are general impressions based on the 
memos and commentaries that could contribute to the final theory 
development. An example of such a theory is "developing the use of 
structures," which is a collection of factors (e.g., the teacher asking children 
to explain their strategy to others, having the children compare a familiar 
structure to an unfamiliar structure that represents the same quantity) that 
the researchers found to appear to be important for stimulating the 
children's learning in this particular instructional setting. The networks that 
the researchers were able to make in ATLAS. ti with connections between 
and within theories, memos, commentaries, and codes are what contributed 
to reaching the top of the hierarchy in analysing this data, namely a local 
instruction theory about the children's spatial structuring ability and the role 
of the instructional sequence in stimulating this ability. 

Similar to the organisation process in the MENS Project, the codes in the 
ToolUse project were initially used to browse through the data and to 
correlate events with the other data sources in ATLAS. ti. First, the data from 
the ToolUse project - screen videos, photocopies of written work, and 
screenshots of students' computer activities - were imported into ATLAS. ti. 
The units of analysis for the clipping of the videos into quotations were the 
exercises that the students did. The quotations in the ToolUse project were 
then coded with the number that corresponded to the activity and the names 
of the student. The researchers started the analysis by watching and 
annotating the work of one pair of students, Lisa and Romy. They first gave 
a short description of the quotation using the comment tool and additional 
analytical notes. These notes then guided the description of a storyline about 
Lisa and Romy's learning processes in terms of the learning environment 
and the relation between tool use and mathematical development (e.g., 
Kieran & Drijvers, 2006; Doorman & Gravemeijer, 2009). 
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Reliability of Data Reduction 

One way in which ATLAS. ti played an important role in contributing to 
the reliability of the MENS and ToolUse projects was allowing the 
researchers to supplement the qualitative data with quantitative output and 
compare the strategies within and across different groups of children. This 
enriched the qualitative description of children's and students' strategies in 
both projects. The quantitative output provided supporting evidence and 
more inspiration for spurring on qualitative analyses. 

The contribution of quantitative output to the qualitative data 
exemplifies Jacobs et al.'s (1999) cyclical analytical process of coding and 
analysing video data. After coding and analysing the videos qualitatively, 
the researchers analysed the frequency counts quantitatively. The 
researchers could then interpret the frequency patterns in light of the 
behavioural patterns that emerged out of their qualitative analyses. This 
concluded one cycle of Jacobs et al.'s analytical process. Yet, at the same time 
it motivated the next cycle because the quantitative outcomes gave rise to 
more research questions that could again be investigated qualitatively. 

Trackability 

In both the MENS and the ToolUse projects, the ATLAS. ti unit that 
contained all the quotations, comments, and codes of the interviews was 
shared amongst project members. The procedure for discussing the results 
from the data analysis was convenient and clear because each quotation 
isolated one question or activity that could be analysed separately from the 
rest of the interview. The discussions helped the project members to come to 
a consensus about the types of strategies that the students used to solve the 
tasks and about the validity and reliability of the instrument itself. This 
interaction contributed to adjusting and improving the classification of the 
typification of strategies and emerging conjectures concerning the learning 
processes. 

In analysing the instruction experiment, it was helpful for the 
researchers to be able to report on conjectured patterns in the teaching and 
learning processes while making direct reference to specific quotations that 
provided supporting evidence. In this way, project members could 
understand the reasoning behind a conjecture, examine the relevant 
quotations and judge the value of the clip or a code as representative of the 
pattern. The acceptances, refutations, and subsequent amendments to these 
conjectures were fundamental to the theory building process. This is what 
lead to defining the commentaries, memos, theories, and, finally, 
preliminary indications for local instruction theories. This process of 
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negotiation illustrates the interaction that is involved in analysing 
qualitative data in design research. 

Organised, Reliable, and Trackable Data for Developing Theory 

To understand the children's learning processes during the instruction 
experiment of the MENS project, it was important to grasp the behavioral 
patterns that occurred as the lessons progressed. The network of codes, the 
different types of memos, and the query tool in ATLAS. ti were tools that 
proved to be essential for elucidating the effects of the instructional sequence 
and the role played by the interaction between the teacher and the children 
as well as amongst the children. The use of ATLAS. ti helped the researchers 
in the MENS project to cultivate the interaction between watching the data 
and evaluating their conjectures. The outcome of one interactive cycle 
stimulated subsequent cycles of analysis that refined coding systems and 
further identified parts of the video that best illustrate particular findings. 
Lor analysing the instruction experiment, the process of coding and recoding 
each of the quotations and building a hierarchy of commentaries, memos, 
and theories made gradually more accessible those behavioural patterns that 
contributed to the development of a local instruction theory about young 
children's spatial structuring. 

In the ToolUse-project, the researchers initially analysed the work of two 
students qualitatively. The idea was that emerging conjectures could be 
tested quantitatively using corresponding codes and the clipped data. The 
two students were expected to use calculation chains in their reasoning 
about input-output-dependencies, and this reasoning was thought to change 
to the use of tables and graphs as basic structuring elements. In line with the 
premises of realistic mathematics education, these expectations were based 
upon the idea of emergent modeling. One of the key aspects of emergent 
modeling is the development of a model of a particular situation into a model 
for ways of mathematical reasoning about situations in general (Gravemeijer, 
2004). Erom analysing the successive lessons during which the two students 
were provided with one tool, it appeared that the manifestation of the input- 
output chain itself did not change, but, instead, that the function of the 
representation changed. As such, the focus of reasoning in the model shifted 
from individual calculations to dependency relationships and dynamic 
processes. 

One outcome of this analysis was that the principle of a form-function- 
shift (Saxe, 2002) could describe what happened in the learning process of 
the two students. The articulation of this shift appeared to be helpful for 
differentiating phases in the learning process with codes and for identifying 
two illustrative items among the activities. The researchers characterised 
different solution strategies using the form-function-shift to construct codes 
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for these items (i.e. from "a record for calculations" to "as a tool for 
reasoning"). The solutions of all the students were coded to investigate 
patterns in these solutions and to find quantitative backing for the emerging 
conjecture. During this coding process, the researchers discussed several 
individual cases to fine-tune the code definitions and their interpretations of 
the codes. The query tool of ATLAS, ti could then be used to make an 
inventory of and to count strategy codes. With that tool it was possible to see 
which students from a specific class used a code at a specific activity (see 
Figure 5). 



Figure 5. An example of a query that shows that 7 students from class "LL- 
Boni" were coded "AP:record of calcs" at activity "dwol-6." 


The analysis offered quantitative evidence for the conjecture of a shift in 
the students' reasoning with the computer tool (Doorman et al., in press). 
This example illustrates how the researchers could analyse individual 
processes as well as check their emerging conjectures quantitatively against 
the work of all the other students. 
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Conclusion 

The MENS and the ToolUse projects that we presented in this paper 
illustrate three ways in which multimedia data analysis software such as 
ATLAS. ti can contribute to cultivating the interaction that is inherent to the 
analysis of qualitative design research data for the development of a local 
instruction theory. First, the organisation of the data itself helps researchers 
to explore the data and consequently begin to build a network of 
commentaries, memos, and theories that can contribute to outlining a local 
instruction theory. Second, supplementing the qualitative data with 
quantitative counts can help to reevaluate initial conjectures and support the 
reliability of the research. Third, the organisation of the data as well as the 
documentation of ongoing analyses stimulates researchers into discussing 
the data with others as a means to ensure the trackability of the data. Hence, 
the process of organising the raw qualitative data in a multimedia data 
analysis program contributes to more than just organised, and therefore 
manageable, data analysis. Rather, it fundaments the quality of the research 
and supports the interaction that is necessary to reevaluate researchers' 
initial conjectures towards developing the local instruction theory. This 
enhances the objectivity and verifiability of the research. 

Both the MENS and the ToolUse projects show how organising the data 
into manageable quotations greatly supported the theory development. The 
multimedia data analysis in the MENS project exemplifies the generation of 
theory about how young children develop spatial structuring ability and 
how this ability may be stimulated in an instructional setting. The ToolUse 
Project shows how its theoretical framework was developed through 
simultaneously analysing the video clips of a pair of students and testing 
emerging conjectures about these clips against the solutions of all the other 
students. 

The researchers in both projects organised the data of instruction 
experiments into coded quotations. This triggered the notation of memos, 
which subsequently inspired the definition of specific commentaries, new 
memos, and broader theories. Further, the counts of codes and quotations 
that resulted from this contributed to improving the reliability of the 
qualitative analyses. The trackability of the research was also supported by 
how conveniently the researchers could present and discuss their findings 
and conjectures by making reference to the quotations, the codes, the 
different types of memos, and the network views. 

The two projects bring into practice the integrated model that we 
proposed earlier in this paper. The interaction between data and analysis, 
which holds Jacobs et al.'s (1999) and Gravemeijer's (2004) models together 
and which propels the research towards the development of an overarching 
local instruction theory, can be recognised on two levels in both projects. 
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First, the interaction between the raw video data and the coding procedure 
as described in Jacobs et al.'s model is illustrated by how the researchers 
coded the quotations for the instruction experiment inductively and 
amended the code book as necessary, depending on further observations of 
the quotations and discussions with project members. 

On a second level, with regard to Gravemeijer's (2004) model, the 
interaction between the students' responses to the first instruction 
experiment, the adjustments to the researchers' conjectures about the 
learning processes, and the successive instruction experiments, is what 
ultimately contributed to the development of the local instruction theory in 
both projects. Importantly, however, the two levels of analysis (i.e., at the 
level of data and at the level of the instruction experiment) are interwoven in 
that the coding of the data at a more practical level fundaments the 
interpretation of the students' responses to the instruction experiment at a 
more theoretical level. This, in turn, influences the researchers' conjectures 
about students' learning processes. As such, the two models lend themselves 
to being integrated into a more encompassing model of theory development 
in design research. 

Despite the benefits of using multimedia data analysis software for 
qualitative design research, it is important to note that an enormous amount 
of time is needed for all the work that is involved in clipping the videos into 
quotations, organising the clips, and developing hypotheses and theories 
about the data. In addition, it is important for researchers to realise that the 
data input in a multimedia data analysis tool will not result in a theory 
output that is a clear-cut answer to the research questions. Instead, the 
benefits of using a multimedia data analysis tool such as ATLAS. ti depend 
on the amount of effort that researchers choose to put into the process of 
data analysis. Hence, researchers must keep a balance between the 
(sometimes tedious) time within which the research has to be completed and 
the time that must be invested in organising the data. 

In this paper we have presented two case studies that show how the 
benefits of using multimedia data analysis software extend beyond merely 
simplifying the complex process of collecting, coding, and analysing raw 
qualitative data. If care is taken to balance the amount of effort that is spent 
organising the data with the time that is left available for actually 
interpreting the data, such software can serve as a valuable tool for 
cultivating the interaction, essential to theory development in design 
research, between watching the videos and evaluating conjectures. We have 
proposed an integrated model that may help mathematics education 
researchers optimise the complex process of analysing qualitative design 
research data for the benefit of developing a thorough, reliable, and well- 
founded local instruction theory. 



28 


Van Nes & Doorman 


Acknowledgments 

The MENS project is supported by a grant from the Netherlands 
Organization for Scientific Research (NWO), project number 051.04.050. It is 
supervised by Prof. J. de Lange of the Freudenthal Institute for Science and 
Mathematics Education and Prof. E. de Haan of the Helmholtz Institute. 

The ToolUse project is also supported by the Netherlands Organization 
for Scientific Research, project number 4121.04.123. The research is 
supervised by Prof. K.P.E. Gravemeijer and the author collaborates with 
Peter Boon, Paul Drijvers, Sjef van Gisbergen, and Helen Reed. 

References 

Anghileri, J. (1989). An investigation of young children's understanding of 
multiplication. Educational Studies in Mathematics, 20, 367-385. 

Battista, M. T., & Clements, D. H. (1996). Students' understanding of three- 
dimensional rectangular arrays of cubes. Journal for Research in Mathematics 
Education, 27, 258-292. 

Battista, M. T., Clements, D. H., Arnoff, J., Battista, K., & Van Auken Borrow, C. 
(1998). Students' spatial structuring of two-dimensional arrays of squares. 
Journal for Research in Mathematics Education, 29, 503-532. 

Cobb, P., diSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in 
educational research. Educational Researcher, 32(1), 9-13. 

Cobb, P., & Yackel, E. (1996). Constructivist, emergent and sociocultural perspectives 
in the context of developmental research. Educational Psychologist, 31, 175-190. 
Doorman, M., Drijvers, P., Boon, P., Van Gisbergen, S., Gravemeijer, K., & Reed, H. 
(in press). Tool use and conceptual development: an example of a form-function 
shift. Paper accepted for publication in the Proceedings of the 33 rd Psychology of 
Mathematics Education conference. 

Doorman, M., & Gravemeijer, K. (2009). Emergent modeling: Discrete graphs to 
support the understanding of change and velocity. ZDM Mathematics Education, 
41, 199-211. 

Drijvers, P., Doorman, M., Boon, P., Van Gisbergen, S., & Gravemeijer, K. (2007). Tool 
use in a technology-rich learning arrangement for the concept of function. In 
D. Pitta-Pantazi & G. Philippou (Eds.), Proceedings of the V Congress of the 
European Society for Research in Mathematics Education CERME5 (pp. 1389 - 
1398). Larnaca, Cyprus: University of Cyprus. 

Freudenthal, H. (1973). Mathematics as an educational task. Dordrecht, the Netherlands: 
Reidel. 

Freudenthal, H. (1983). Didactical phenomenology of mathematical structures. Dordrecht, 
the Netherlands: Reidel. 

Freudenthal, H. (1991). Revisiting mathematics education: China lectures. Dordrecht, the 
Netherlands: Kluwer Academic Publishers. 

Glaser, B., & Strauss, A. (1967). Discovery of grounded theory. Chicago: Aldine. 
Gravemeijer, K. (1994). Developing realistic mathematics education. Utrecht, the 
Netherlands: CD~p Press. 



Multimedia Data Analysis and Theory Development in Design Research 


29 


Gravemeijer, K. (1998). Symboliseren en modelleren als wiskundige activiteit. In N. 

Boswinkel & M. Dolk (Eds.), Over Rekenen Gesproken - Taal in/en Rekenen (pp. 

35-51). Utrecht, the Netherlands: Panama/ Freudenthal Institute. 

Gravemeijer, K. (2004). Local instruction theories as means of support for teachers in 
reform mathematics education. Mathematical Thinking and Learning, 6, 105- 128. 

Gravemeijer, K., Bowers, J., & Stephan, M. (2003). A hypothetical learning trajectory 
on measurement and flexible arithmetic. In M. Stephan, J. Bowers, P. Cobb, & K. 
Gravemeijer (Eds.), Supporting students' development of measuring conceptions: 
Analyzing students' learning in social context (pp. 51-66). Reston: NCTM. 

Gravemeijer, K., & Cobb, P. (2006). Design research from the learning design 
perspective. In J. van den Akker, K. Gravemeijer, S. McKenney, & N. Nieveen 
(Eds.), Educational design research: The design, development and evaluation of 
programs, processes and products (pp. 45-85). London: Routledge. 

Jacobs, J. K., Kawanaka, T., & Stigler, J. W. (1999). Integrating qualitative and 
quantitative approaches to the analysis of video data on classroom teaching. 
International Journal of Educational Research, 31, 717-724. 

Kieran, C., & Drijvers, P. (2006). The co-emergence of machine techniques, paper- 
and-pencil techniques, and theoretical reflection: A study of CAS use in 
secondary school algebra. International Journal of Computers for Mathematical 
Learning, 1 1(2), 205-263. 

Mulligan, J. T., Mitchelmore, M. C., & Prescott, A. (2005). Case studies of children's 
development of structure in early mathematics: A two-year longitudinal study. 
In H. Chick & J. Vincent (Eds.), Proceedings of the 29th annual conference of the 
International Group for the Psychology of Mathematics Education (Vol. 4, pp. 1-8). 
Melbourne: Psychology of Mathematics Education. 

Mulligan, J., Mitchelmore, M., Martson, J., Highfield, K., & Kemp, C. (2008). 
Promoting mathematical pattern and structure in the first year of schooling: An 
intervention study. Paper presented at the 32nd Annual Conference of the 
International Group for the Psychology of Mathematics Education. Morelia, 
Mexico: Psychology of Mathematics Education 

Pitta-Pantazi, D., Gray E., & Christo u, C. (2004). Elementary school students' mental 
representations of fractions. In M. Homes & A. Fuglestad (Eds.), Proceedings of 
the 28th annual conference of the International Group for the Psychology of 
Mathematics Education (Vol. 4, pp. 41-48). Bergen, Norway: Bergen University 
College. 

Powell, A. B., Francisco, J. M., & Maher, C. A. (2003). An analytical model for 
studying the development of learners' mathematical ideas and reasoning using 
videotape data. Journal of Mathematical Behavior, 22, 405-435. 

Saxe, G. B. (2002). Children's developing mathematics in collective practices: A 
framework for analysis. The Journal of the Learning Sciences, 11, 275-300. 

Schoenfeld, A. H. (2000). Purposes and methods of research in mathematics 
education. Notices of the AMS, 47, 641-649. 

Simon, M. (1995). Reconstructing mathematics pedagogy from a constructivist perspective. 

Journal for Research in Mathematics Education, 26, 114-145. 

Smaling, A. (1987). Methodologische Objectiviteit en Kwalitatief Onderzoek 
[Methodological objectivity and qualitative research], Lisse, the Netherlands: 
Swets & Zeitlinger. 



30 


Van Nes & Doorman 


Stigler, J. W., Gallimore, R., & Hiebert, J. (2000). Using video surveys to compare 
classrooms and teaching across cultures: Examples and lessons from the TIMSS 
video studies. Educational Psychologist, 35, 87-100. 

Strauss, A. & Corbin, J. (1998). Basics of qualitative research. Techniques and procedures 
for developing grounded theory (2nd ed.). London: Sage Publications Inc. 

Van Eerde, H. A. A. (1996). Kwantiwijzer: Diagnostiek in reken-wiskundemderwijs 
[Kwantiwijzer: Diagnostics in mathematics education]. Tilburg, the Netherlands: 
Zwijsenbv. 

Van Nes, F. & De Lange, J. (2007). Mathematics education and neurosciences: 
Relating spatial structures to the development of spatial sense and number 
sense. Tlw Montana Mathematics Enthusiast, 4(2), 210-229. 

Van Nes, F. & Doorman, M. (2006). Mathematics education and neurosciences. 
PanamaPost, 25(4), 3-10. 

Van Nes, F. & Doorman, M. (2009). Miertje Maniertje en de Maniertjesdoos [Ant and 
its tool box], Tijdschrift voor Nascholing en Onderzoek van het Reken- 
wiskundeonderwijs, 28(1), 62-71. 


Authors 

Fenna van Nes, Freudenthal Institute for Science and Mathematics Education, 
Ai'dadreef 12, 3561 GE Utrecht, The Netherlands. <fennan@fi.uu.nl>. 

Michiel Doorman, Freudenthal Institute for Science and Mathematics Education, 
Ai'dadreef 12, 3561 GE Utrecht, The Netherlands. <m.doorman@fi.uu.nl>. 




